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into the curriculum. Findings from the evaluation components are explored, 
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learned from this project, guidelines are suggested for those who may be 
interested in a similar undertaking. A student survey form is appended. 
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The Multiple Challenges of Multimedia: Development , Implemen- 
tation , and Evaluation 

MARGARET ANN KASSEN, The Catholic University of America 

This article presents an overview of an ongoing initiative to develop, in- 
tegrate, and evaluate multimedia lessons to enhance listening compre- 
hension in Spanish language classes at The Catholic University of Amer- 
ica. In an effort to focus on the process involved, the two phases that 
have been completed are examined, including the rationale for the proj- 
ect, the selection of the Libra authoring system, the building of the 
authoring team, the design and revision of the lessons, and their integra- 
tion into the curriculum. Findings from the evaluation components are 
explored with particular emphasis on the evidence pointing to the need 
to understand better how learners actually interact with multimedia. 
Based on the lessons learned from this project, guidelines are suggested 
for those who may be interested in pursuing a similar undertaking. 

INTRODUCTION 

In the second language teaching field today, there is great interest in 
technology and its potential to enhance second language learning. Multime- 
dia has a particular appeal, due in part to what Pusack and Otto (1997) refer to 
as its "capacity to access and control via computer a full range of familiar me- 
dia: text, motion video, photo images, sound and graphics" (p. 2). Many of the 
materials teachers have previously used effectively to promote language 
learning can be placed at die learners' fingertips in the digital environment 
with the promise of increased learning outcomes. The allure of multimedia 
must be tempered, however, with the realities that confront any instructional 
innovation: the challenges of development, implementation, and evalua- 
tion. How does one go about developing multimedia lessons for use in mul- 
tiple-section courses? How can these lessons be integrated in the existing cur- 
riculum? How can multimedia be evaluated? These fundamental questions 
will be examined in the context of a multimedia project for beginning level 
Spanish undertaken at The Catholic University of America (CUA). 

BACKGROUND OF THE PROJECT 

The Department of Modem Languages at CUA was interested in explor- 
ing language learning technology, particularly as a way to enhance students' 
out-of-class language learning. This option had become feasible due to our re- 
cently installed Multimedia Language Center, a small, independent study 
room equipped with computers, VCRs, laserdisc, etc. As a first step, the media 
director and language coordinator attended a workshop on Libra, a multime- 
dia authoring system designed to facilitate listening comprehension. Libra 
was developed by Robert Fischer and Mike Farris at Southwest Texas State 
University at San Marcos [1] with the support of the Fund for the Improve- 
ment of Post-Secondary Education (FIPSE). Because the listening comprehen- 
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sion goals of CUA's Spanish pro- 
gram Destinos were compatible with 
those of Libra, the team applied for 
and received a grant to develop, im- 
plement, and evaluate Libra lessons 
for Destinos as a part of Fischer's 
FIPSE project. 

LISTENING COMPREHENSION 
AND TECHNOLOGY 

A focus on listening compre- 
hension at the novice and interme- 
diate levels is well supported by re- 
search and theoretical models that 
point to comprehensible input and 
intake as fundamental to second 
language acquisition. As Rubin 
(1994) noted in her review of the 
relevant literature, the actual process 
of listening has received consider- 
able research attention. Comprehen- 
sion has come to be seen as a dy- 
namic, interactive, cognitive process 
in which listeners construct mental 
representations of their understand- 
ing of texts, using and coordinating 
both concept-driven (top-down) and 
sentence and word-level (bottom- 
up) processes. Furthermore, in real- 
world, face-to-face communication, 
listeners have immediate access to 
information about the context of 
their interactions (participants and 
setting) and paralinguistic cues (such 
as eye movement and gestures), and 
they generally have some control of 
the interaction by means of negotia- 
tion and use of strategies (Joiner, 
1997). 

Given this understanding of 
the listening process, how can tech- 
nology best be brought to bear on lis- 
tening skill development? While 
traditional cassettes provide the 
audio component and video the 
visual information, these media do 
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not facilitate the interaction of 
learners with the text other than 
through their cumbersome replay 
option. Multimedia, however, be- 
cause of its digital nature, allows di- 
rect access to and manipulation of 
particular segments of text and, by 
means of various help features, of- 
fers the user added options for man- 
aging the interaction. As Joiner 
(1997) notes. 

Computer-assisted multimedia 
comes closer than the other audio 
and visual media to meeting the 
standard for listening embodied 
in face-to-face communication. 
Presence, interactivity, control, 
multisensory input, and multiple 
sources of assistance can be incor- 
porated into this sophisticated 
technology, (p. 90) 

LIBRA AUTHORING SYSTEM 

Libra is a hypercard-type program for 
Macintosh [2] that allows the teacher 
to create a stack of question cards 
written to target selected segments of 
a videotext. These question cards ap- 
pear on the computer screen and are 
linked to video clips that have either 
been digitized and are accessible on 
the same screen or that are shown 
on a laserdisc player, as was the case 
in this study. The Libra documenta- 
tion recommends beginning each 
lesson with a textmap to acquaint 
learners graphically with the struc- 
ture of the text. These advance or- 
ganizers may reappear before each 
segment to lead users from one 
segment to another logically (see 
Figure 1). The five question tem- 
plates available in the program are 
flexible and may be used to focus 
learners' attention on various as- 
pects of listening comprehension. 
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from main ideas to details and spe- 
cific words. For multiple choice, 
checklist, binary checklist, and icon 
sorting questions (see Figures 2, 3, 4, 
and 5), learners select the correct re- 
sponses by clicking or dragging and 
get immediate feedback. The fifth 
question type is open-ended and al- 
lows users to create their own an- 
swers by typing into a notebook. 
While the teacher sets parameters by 
deciding which portion of the video 
is available on a given card, learners 
are given considerable control of 
their learning: they may rewind, re- 
play, and fast-forward as needed, and 



they may access a number of help 
features such as a dictionary, a list of 
characters, a story summary, script, 
and special notes. 

THE PROJECT 

The multimedia project at CUA 
has gone through two phases, each 
of which involved developing, im- 
plementing, and evaluating lessons. 

Phase 1: Development and Imple- 
mentation 

In the initial pilot study in the 
spring of 1995, the language coordi- 
nator/media director team and a 




LA EXCAVACION 



Esta escena nos relata lo que Arturo esta haciendo en el Distrito 
Federal de Mexico mientras espera una llamada de Raquel. Cliquea 
el icono en la caia para continuar. . 



Figure 1. Advance Organizer 
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La excavacion, pregunta 1 



Si quieres, utiliza el control del video para repasar la escena. 
^En que pensaba Angela? 



O Pensaba en que si Roberto tenia hambre. 

O Pensaba en que si Roberto tenia suficiente aire. 
O Pensaba en que si Roberto estaba enfermo. 



Click here for feedback. 



Move mouse over buttons for an explanation. 



Figure 2. Multiple Choice Questions 



Esperando: pregunta 3 




IJriliza el control de video para repasar la escena v contesta la pregunta sobre 


Raquel. 




Cliquea el boton a la derecha para escuchar la pregunta. 


B 


Escoge todas las respuestas correctas. 


pregunta 


□ medica 

□ abogada 

□ veterinaria 

□ actriz 

□ ama de casa 

□ profesora 




0 




Click here lor feedback. 




Move mouse over 


buttons for an explanation. 








Figure 3. Checklist Questions 
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La civilization maya: pregunta 4 



Para contestar la pregunta, repasa la escena usando el control del video 
Marca las respuestas como cierta o falsa. Los mavas ... 




derta falsa 



□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

0 

Click here for feedback. 



gobemaban un vasto imperio del este al oeste. 
vivian en la peninsula del Yucatan, 
formaban una serie de estados autonomos. 
llegaron a su maximo apogeo en el tiempo de Cristo. 
hideron grandes a vances en el campo de la filosoffa. 
sabian todos los ciclos de los eclipses solares y lunares 



Move mouse over buttons for an explanation. 



Figure 4. Binary Checklist Questions 



Ar turo en Mexico, pregunta 3 



mm 



Mira la siguiente escena: 
^Adonde va Arturo? 



bar 






gasolinera 



almacen banco 

• • 

x m 

cine supermercado 



farmacia 




Click here for feedback. 



Move mouse over buttons for an explanation. 



Figure 5. Icon Sorting Questions 
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graduate teaching assistant (TA) de- 
signed Libra lessons of approxi- 
mately 20 cards each to accompany 
eight Destinos episodes used in sec- 
ond- and third-semester Spanish. 
The 25-minute episodes were seg- 
mented into scenes, and text maps 
led the learners to view each scene 
in its entirety without stopping. Sub- 
sequent cards asked questions on 
both main ideas and details in the 
scene and provided learners the op- 
portunity to review relevant por- 
tions of the scene as needed. Due to 
time limitations, no glossary was 
provided, and scripts and character 
lists were not available for all les- 
sons. 

At the second-semester level, 
two instructors of four class sections 
with a total of 59 students partici- 
pated in Phase 1. The third-semester 
group involved three instructors of 
six sections with 106 students. For 
each lesson, students had one week 
to go to the Multimedia Language 
Center individually or with a part- 
ner to complete the assignment. 
They either completed the Libra les- 
son linked with the Destinos vide- 
odisc in the experimental group, or 
they viewed the video and filled in a 
written worksheet in the control 
group. The questions in both condi- 
tions were substantially the same, 
though some of the written ques- 
tions had to be adapted from elec- 
tronic to print medium. The instruc- 
tors were requested not to discuss 
the episode until the day it was to be 
completed. 

Because the episodes assigned 
for out-of-class work were an inte- 
gral part of the Destinos program, 
the project team felt that they would 

O 
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integrate smoothly into the class- 
work just as the other videos viewed 
in the lab did. To reinforce the no- 
tion that these assignments were in- 
deed required, the syllabus listed 
their due dates, and the course de- 
scription stated that they counted as 
10% of the final course grade. 

Phase 1: Evaluation 

The evaluation of the pilot 
study was based on a time-series ex- 
perimental design in which the stu- 
dents alternated between the ex- 
perimental Libra condition and the 
control group. A modified recall pro- 
tocol pretest served to establish pre- 
existing differences among the stu- 
dents. Similar recall protocols in 
English, collected in class the day 
students were to have completed 
viewing the assigned episode, served 
as the posttest. Due primarily to an 
uncooperative instructor, the sec- 
ond-semester data were quite in- 
complete and yielded no statistically 
significant results. The third- 
semester data were reduced by stu- 
dent absences, but complete data sets 
were available on 17 students in the 
experimental group and 20 in the 
control group. An analysis of covari- 
ance revealed that on two of the four 
posttests, the experimental group 
significantly outperformed the con- 
trol group. A third posttest ap- 
proached significance (see FIPSE, 
1994, for complete data and explana- 
tion). Thus, despite the limited data, 
the pilot study provided evidence of 
the positive impact that the Libra- 
based viewing of the Destinos videos 
had on comprehension. 

Student questionnaires admin- 
istered at the end of the semester 
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complemented the quantitative 
analysis. These questionnaires re- 
vealed a generally positive attitude 
toward the multimedia lessons (see 
Table 1). The areas of concern identi- 
fied by the student users included 
the length of time required to com- 
plete the lessons and the lack of un- 
derstanding of the program on the 
part of some of the instructors, stu- 
dents, and lab attendants. In its own 
evaluation of the project, the devel- 
opment team posited two additional 
factors that may have contributed to 
the lack of regular student participa- 
tion: insufficient integration into 
classwork and the redundancy built 
into the Destinos program, which, 
appropriately for a soap opera, al- 
lows viewers to follow the story line 
even if they miss an episode. 



Phase 2: Development and Imple- 
mentation 

The second phase of the project 
focused on addressing the concerns 
raised in the pilot and on improving 
participation. The Phase 1 authoring 
team and two additional TAs recon- 
figured four of the lessons for sec- 
ond-semester Spanish (see sample 
cards in Figures 2 to 5). With the 
continued support of the FEPSE proj- 
ect, one of the TAs had received a 
grant to attend a Libra workshop, 
and he then helped teach the 
authoring system to the other TA 
team member as they rewrote the 
lessons. 

Two major changes were incor- 
porated into the lesson revisions. 
First, given the redundant nature of 
the videos, the team reduced the 



Table 1 

Attitude Questionnaire Item from Phase 1 (n=80) 



My attitude toward using computers to learn a foreign language: 





Before This Semester 


After This Semester 


Hostile 


i 


2 


Hesitant 


13 


14 


Indifferent 


30 


22 


Curious 


31 


21 


Enthusiastic 


5 


21 
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amount of video to be viewed in 
each lesson to three to four seg- 
ments: one or two that presented the 
key story line events, one that fo- 
cused on culture, and one that tar- 
geted functional language use. Sec- 
ond, several adjustments were made 
to the lessons to allow the learner 
greater flexibility in interacting with 
the text. A video control panel was 
added to every card so students 
could fast-forward, rewind, and re- 
play as needed at any time. More 
help features, in the form of scripts, 
character lists, and a glossary, were 
provided in three of the four les- 
sons. A third change involved the 
addition of video-related grammar 
exercises, but because this compo- 
nent was not included in the pilot 
study, it will not be examined in this 
article. 

Significant changes were made 
in the implementation of the re- 
vised lessons. The two second- 
semester Spanish instructors were 
closely involved in the develop- 
ment of the lessons, and, as a result 
of their investment in the process, 
they understood Libra clearly and 
were able to explain it adequately to 
their students. The instructor team 
made a conscious effort to integrate 
the viewings more tightly with 
coursework. They agreed to hold de- 
tailed discussions of the targeted 
scenes in class, to give brief dicta- 
tions and quizzes on them, and to 
incorporate them directly into exam 
questions. Care was also taken to 
train the lab assistants more thor- 
oughly on the program. 

During the spring semester of 
1997, the 55 students enrolled in the 
four sections of second-semester 
Spanish taught by the two TAs who 

0 




helped revise the Libra lessons par- 
ticipated in Phase 2 of the project. As 
requested by the students in the pilot 
study and recommended by Masters- 
Wicks et al. (1996), an orientation to 
the program was held in class before 
the first lesson was assigned. Using 
transparencies of sample computer 
screens, the instructors explained the 
question types and the general navi- 
gation features. As before, the les- 
sons represented 10% of the final 
grade, and due dates were listed on 
the syllabus. 

The final evaluation, which 
was qualitative in nature, was done 
via a user questionnaire (see the ap- 
pendix). The questionnaire included 
Likert-scale items to assess student 
reactions and open-ended questions 
to gain more insight into the stu- 
dents' own perceptions of the les- 
sons. Instead of waiting until the 
end of the semester to assess stu- 
dents' views, the students completed 
the Phase 2 questionnaire in the 
Multimedia Language Center fol- 
lowing completion of the last lesson. 
It was hoped that, by answering the 
questions immediately after the les- 
son, learners would provide more 
accurate information because their 
experiences would be fresh in their 
minds. 

Phase 2: Evaluation Results and 
Discussion 

A total of 30 students out of the 
55 enrolled completed the question- 
naire. The instructors were surprised 
at this low response rate because 
more than 30 students reported do- 
ing the Libra lessons. Some students 
may have done all the lessons except 
the final one when the question- 
naire was distributed. Perhaps the 
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lab attendants were not diligent 
enough in handing out and collect- 
ing the questionnaires as students 
completed the lesson. While the 
causes cannot be determined with 
certainty, the 55% response rate is 
evidence of the difficulty of investi- 
gating a phenomenon that is essen- 
tially an out-of-class, independent 
activity. 

The four Likert-scale questions 
are presented in Table 2. The re- 
sponses to Questions 1 and 2 suggest 
that this second phase more success- 
fully provided the support learners 
needed to feel comfortable using the 
computer materials. The third and 
fourth questions again reflect a gen- 
erally positive (though not enthusi- 
astic) attitude toward computer- 
mediated learning: fewer students 



were negative at the end of the proj- 
ect than at the beginning. 

The following five open-ended 
questions composed the second part 
of the survey: 

1. What was the main benefit you 
derived from doing the Libra 
computer lessons? 

2. What strategies did you use to 
maximize this benefit? 

3. What did you like the most 
about doing the Libra computer 
lessons? 

4. If you could improve one aspect 
of the Libra computer lesson, 
what would it be? 



Table 2 

Likert-Scale Questionnaire Items from Phase 2 (n=30) 





SA 


A 


N 


D 


SD 


1 . I had sufficient orientation to the Libra program 
before doing the lessons on my own. 


10 


9 


3 


6 


1 


2. The lab attendants are helpful to me when I 
use the multimedia program. 


8 


13 


5 


2 


0 


3 . At the beginning of the semester, my attitude 
toward using computers to learn a foreign lan- 
guage was positive. 


8 


8 


10 


4 


0 


4. After the completing the Libra computer les- 
sons, my attitude toward using computers to 
learn a foreign language is positive. 


8 


13 


8 


1 


0 



SA = Strongly Agree, A=Agree, N=No Opinion, D=Disagree, 
SD=Strongly Disagree 
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5. In your opinion, what would be 
the ideal number of Libra com- 
puter lessons to do [per semes- 
ter]? How much time would you 
be willing to spend on each? 

While there was considerable 
variation in the wording of individ- 
ual student responses, the answers 
to these questions were revealing, in 
particular when considering the pat- 
terns that emerged across student 
comments. Tables 3, 4, and 5 sum- 
marize the responses to the first 
three questions, including the cate- 
gory or pattern of response, the 
number of students who made 
comments with that focus, and some 
sample comments. 



When questioned about their 
perceptions of the benefits of the Li- 
bra lessons (see Table 3 for sum- 
mary), half of the students (14) ex- 
pressed the view that the lessons 
helped them "hear" or "understand" 
the videos. This response is in line 
with the listening comprehension 
goals of the authoring system itself 
and our own programmatic objec- 
tives and provides encouraging evi- 
dence of Libra's face validity. Closely 
related to the comprehension focus 
was the observation of three stu- 
dents that the lessons emphasized 
the storyline. This awareness of 
main events appears to support the 
use of the textmap as a way to offer 
learners support in building the 



Table 3 

Questionnaire Responses, Phase 2: Student-Perceived Benefit of 
Multimedia Lessons (n =30) 



Question 1. What was the main benefit you derived from doing the Libra 
computer lessons? 



Category of Response 


Number of 
Responses 


Sample Comment 


Listening comprehension 


14 


• To see and hear lesson at my speed 

• Understanding more of what was said 






by the characters 


Focus on content 


3 


• Reinforced the storyline and made it 






more clear 


Practice 


3 


• To watch a lesson and do exercises 






about it 


Vocabulary 


2 


• I gained a larger vocabulary 




12 
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macropropositional structure of the 
text. Additionally, three students 
identified practice as Libra's greatest 
benefit, an apparent recognition of 
being more actively involved than 
in traditional video viewing. Fi- 
nally, two students specified that Li- 
bra was useful in improving their 
vocabulary. While no vocabulary 
assessment was done in this study, 
Chun and Plass (1996) found evi- 
dence of considerable vocabulary 
learning in their work with multi- 
media to enhance reading compre- 
hension. Their work and student 
comments in this study raise ques- 
tions about the possibility of inciden- 
tal vocabulary learning in the Libra 
multimedia environment, questions 
that need to be addressed in future 
research. 

The second question asked stu- 
dents to reflect on and identify the 



strategies they had used while doing 
the Libra lessons (see Table 4). Most 
frequently cited (by ten students) was 
the use of the rewind/ replay button, 
with comments such as "I listened to 
segments more than once using re- 
view buttons." Students' willingness 
to interact with the computer in or- 
der to adjust for their own learning 
needs is an example of the positive 
side of learner control in multime- 
dia. 

A second strategy mentioned by 
three students was attending more 
closely to the material; as one stu- 
dent said, "[I] listened to it more 
than I pay attention to Destinos." 
Multimedia lessons written with 
systems such as Libra have the po- 
tential to facilitate attention in a 
number of ways: video is segmented 
and efficiently accessible, lessons are 
task-oriented, earphones reduce 



Table 4 

Questionnaire Responses, Phase 2: Strategy Use (n=30) 



Question 2. What strategies did you use to maximize this benefit? 



Category of Response 


Number of 
Responses 


Sample Comment 


No response 


12 




Replay 


10 


• Using the rewind button to review when 
confused or unsure 


Attention 


3 


• Listened to it more than I pay attention 
to Destinos 


Visuals 


2 


• Watching the videos themselves 




13 
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sound distortion, and students 
physically interact with the com- 
puter via mouse clicks to progress 
through the lesson. Because paying 
attention is one of the metacognitive 
strategies considered essential for 
language learning (Oxford, 1990), 
this capacity may be one of multi- 
media's important strengths. 

In a response that may be re- 
lated to learning style preferences, 
two students claimed to attend to the 
videos themselves. Since the visual 
stimulus helps to contextualize the 
input, focusing on the images may 
be a particularly helpful strategy to 
visual learners. Though learning 
styles were not examined in this in- 
vestigation, there is growing re- 
search interest in this area (Beauvois 
& Eledge, 1996; Chun & Plass, 1996; 
and Meunier, 1997). Other strategies 
mentioned (each by one student) in- 
cluded doing the lessons with a 
partner, reading the questions in ad- 
vance, and consulting the feedback. 

Perhaps most notable is what 
students did not say in response to 
this question. Twelve of the 28 stu- 
dents did not offer any strategies at 
all. Though the lack of comment 
does not necessarily mean that the 
students did not make use of any 
strategies, it does raise a fundamen- 
tal question: Are students suffi- 

ciently aware of their learning needs 
in order to address them? In the tra- 
ditional, teacher-mediated class- 
room, metacognitive strategies that 
enable learners to address their 
learning needs are essential, but they 
may be even more critical in the 
autonomous learning environment 
of multimedia. 



As a case in point, the ques- 
tionnaire results suggest that the 
students may not have taken full 
advantage of some of the help fea- 
tures. While 80 to 90% of the stu- 
dents claimed to use the replay func- 
tion "sometimes" or "often," 25% of 
the students reported "never" using 
the script or glossary, 40% reported 
using them "sometimes," and the 
remainder did not respond. The un- 
even use of help functions may be in 
part due to issues such as the level of 
difficulty of the lesson questions, the 
screen design of these features, the 
lack of adequate training on their 
use, and forgetfulness on the part of 
the students. Though the version of 
Libra used in this project did not al- 
low for tracking student actions [3], 
studies that examined computer logs 
for computer-assisted writing (Scott, 
1990) and reading (Davis & Lyman- 
Hager, 1997) came to a similar con- 
clusion: students did not make ade- 
quate use of the resources provided. 
Based on their findings with third- 
semester French students, Davis and 
Lyman-Hager suggest that "students 
need training to use computers in 
the most beneficial way" (p. 68). 

What are the strategies that 
"good multimedia learners" use? 
Can or should these strategies be 
taught? If so, how? How is strategy 
use affected by proficiency level and 
motivation? Hoven (1997) proposes 
that not all second language learn- 
ers, especially in the initial stages of 
their learning, want or are able to 
take control (105). She advocates a 
hierarchy of task types to raise stu- 
dents' awareness of strategies. As 
recommended by Pusack and Otto 
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(1997), "issues of control and interac- 
tivity in multimedia instruction ur- 
gently require more attention from 
researchers to inform development" 

(p. 10). 

When asked what they liked 
best about the multimedia lessons, 
students responded quite pragmati- 
cally (see Table 5). Nine students in- 
dicated that the lessons helped them 
achieve their short-range goals ("It 
helped my understanding of Desti- 
nos ") and their longer- range goals 
("Improved my listening skills," 
"Very helpful on tests"). Five stu- 
dents highlighted the learner- 
centeredness of the lessons, for ex- 
ample, being able to "choose [their] 



own time [to do the lessons]," 
"partner up", and "start[ing] and 
stop [ping] when I need to." 

The ability to check their work 
via the feedback button was cited as a 
plus by four students. In the litera- 
ture, access to immediate, non- 
threatening feedback is widely seen 
as an advantage of computer-assisted 
learning (Bush, 1997; Pennington, 
1996; Pusack & Otto, 1997). In the 
foreign language learning context, 
the research on error correction has 
been focused primarily on oral feed- 
back, and results have been mixed 
(for a summary, see Chaudron, 
1988). As yet, there is little research 
on the impact of computer-provided 



Table 5 

Questionnaire Responses, Phase 2: What Students Liked the Most About the 

Lessons (n =30) 



Question 3. What did you like the most about doing the Libra computer 

lessons? 



Category of Response 


Number of 
Responses 


Sample Comment 


Goal-oriented 


9 


• Made the lesson more dear 






• Very helpful on tests 

• Improved my listening skills 


Control 


5 


• Having the option of starting and stop- 
ping when I need to 

• Choose own time 

• Can partner up if you want 


Feedback 


4 


• Checking your answers 
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feedback on language learners. How 
do learners react to computer feed- 
back? Is it anxiety-producing? What 
types of feedback are most helpful? 

In a study of feedback on com- 
puter-assisted grammar practice, Na- 
gata and Swisher (1995) found evi- 
dence that 'smart' feedback, which 
gave further information about er- 
rors, was more useful to students 
than the generic variety. What feed- 
back is most appropriate for com- 
prehension-based tasks? The feed- 
back provided in the Libra lessons 
was generic, offering such responses 
as the following: "All of the answers 
so far are correct. Keep going." 
"None of your answers are correct." 
"Some of your answers are correct 
and some are not." Despite the gen- 
eral nature of this feedback, 68% of 
the students claimed to use the feed- 
back "sometimes" or "often," and 
68% described it as "helpful" or 
"very helpful." There were, how- 
ever, two requests for improved 
feedback. One student noted, "If I got 
it wrong, I never knew why." The 
lack of understanding of this poten- 
tially valuable component of the 
multimedia environment points 
clearly to a need for further research. 

The final questions provided 
input for further cycles of develop- 
ment and implementation. For ex- 
ample, one student recommended 
showing the video on the same 
screen as the questions, a possibility 
with newer versions of the Libra 
program, but feasible only with the 
addition of a video server to store 
the lesson segments. The generally 
positive attitude of the students to 
the Libra lessons was reflected in 
their responses to the last question. 
Three students wanted to do one 



multimedia lesson for each of the 17 
chapters studied during the semes- 
ter. Others offered suggestions rang- 
ing from 3 to 6 lessons, with 5 as the 
median. Almost half of the students 
expressed a willingness to spend 
from 20 to 40 minutes per lesson. At 
the present time, the Libra lessons 
that were not updated in Phase 2 are 
currently being restructured to be 20 
to 30 minutes in length. 

CONCLUSION AND 
IMPLICATIONS 

The effective use of multimedia 
presents challenges at many levels. 
For those who are interested in ex- 
ploring this instructional tool, sev- 
eral recommendations follow from 
the CUA experience. 

1. Select authorware according to 

its theoretical foundation and 

your project goals. 

As the array of possibilities for 
authorware is growing rapidly, the 
selection process needs to be made 
with care. Authorware should be se- 
lected according to its compatibility 
with current language learning 
models and the project goals. Libra 
was created with comprehension 
theory in mind and our goals were 
listening comprehension. Other 
programs such as GALT and An- 
notext are designed to facilitate read- 
ing comprehension (for a summary 
of currently available authorware by 
language skill, see Martinez & Her- 
ren, 1998). 

2. Create broad support for your 

project. 

Due to the time, skill, and en- 
ergy demands of working with mul- 
timedia, broad support is necessary. 
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The development team might in- 
clude the language coordinator, an- 
other faculty member(s), the media 
director, graduate teaching assis- 
tants), and undergraduate assistants. 
Outside support from the software 
creator and grant project director is 
necessary for technical reasons and 
can also be valuable in building sup- 
port for the project at higher levels 
in the university. Backing from the 
department chair and dean is impor- 
tant as release time and funds are 
necessary for project members to at- 
tend workshops and develop and 
rework lessons. This high-level sup- 
port is also key when faculty submit 
multimedia projects as part of their 
activity reports and tenure and pro- 
motion reviews. 

3. Assess facilities and technical 
issues in advance. 

Because university facilities, 
academic computer support, and 
software requirements vary widely, 
it is not useful to outline specific 
technical guidelines. The basics of 
availability and compatibility of 
equipment and software need to be 
addressed, however, as do logistical 
considerations, such as the follow- 
ing: How many stations are available 
for how many students? Should 
students reserve time slots? Should 
they work in pairs? Are the lessons 
properly loaded and easily accessible 
on each station? Are lab assistants 
familiar enough with the lessons to 
help students as needed? Dealing 
with these issues will help the proj- 
ect run more smoothly. 



4. Familiarize instructors with the 

program and the project. 

Instructor preparation is crucial 

to the implementation process. Not 
only do instructors need to be famil- 
iar with how the program functions, 
but they also need an understanding 
of the rationale for the program in 
order to respond adequately to stu- 
dents' questions. It may be useful to 
acquaint new instructors with the 
history of the project and, if possible, 
to involve them in revising ques- 
tions. In this manner, instructors 
will feel more a part of the lessons 
and will be able to offer useful sug- 
gestions for future improvements. 

5. Integrate the out-of-class work 

into class. 

Integrating the lessons into the 
coursework is a substantial chal- 
lenge. Including the project on the 
course description and assigning it a 
percentage of the grade are only the 
beginning. Conducting student ori- 
entations helps make student users 
more aware of what they are ex- 
pected to do and allows them to be- 
gin to establish a comfort level with 
the lessons. Beyond the need for 
these preliminary arrangements, our 
experience points to one key reality 
of the use of multimedia, especially 
for out-of-class work: learners inter- 
act variably with the program. Some 
students may do the lessons quickly 
with no difficulties, some may listen 
several times to the segments and 
consult numerous help features, 
others may begin but not finish, and 
others may not do the lessons at all. 




17 



166 Texas Papers in Foreign Language Education 



Because the use of multimedia is a 
fragmentary experience that differs 
from one learner to another, spend- 
ing time in class to recreate and 
unify the experience for the class as a 
whole is indispensible [4]. Thus, fol- 
low-up activities such as putting 
events in order, identifying charac- 
ters and their roles, and summariz- 
ing, as well as providing traditional 
matching and dictation quizzes, 
need to be incorporated. Exams can 
also include questions that focus on 
the lesson segments. 

6. Include evaluation. 

Evaluation is an essential com- 
ponent of the process. The informa- 
tion it provides supports the subse- 
quent development and implemen- 
tation cycles and brings new ques- 
tions into focus. Both phases of this 
project experienced difficulties with 
two overlapping concerns: participa- 
tion and data gathering. Actual stu- 
dent participation in the lessons, 
which cannot be documented due to 
the lack of computer tracking, is not 
clearly reflected in the data collected. 
In addition to using the computer 
logs now available in Libra, en- 
hancements of class integration 
techniques, improved lab attendant 
oversight, and increased access to 
videodisc stations to accommodate 
student schedules better may result 
in greater participation and better 
data in future phases. It may also be, 
however, that not all learners, par- 
ticularly those at the first and second 
year levels, are motivated enough to 
do multimedia lessons outside of 
class. Investigating the use of mul- 
timedia with more advanced, more 
motivated students might result in 



increased participation and more 
complete data [5]. 

A fundamental issue revealed 
in this project is how little is actually 
known about what learners do with 
multimedia. The limited number of 
strategies reported and the number 
of students who listed no strategies 
suggest a lack of awareness of how to 
learn with multimedia. While 
much work has been done in identi- 
fying strategies in communication, 
research needs to identify those that 
are appropriate for computer- 
mediated learning. In addition to 
computer tracking of students' inter- 
actions with computers, researchers 
need to collect retrospective proto- 
cols to gain a better understanding of 
the strategies used. This approach 
can be incorporated in future cycles 
of evaluation through the random 
selection of a subset of participants to 
monitor more closely via appoint- 
ments for the lessons and for follow- 
up interviews. 

Other areas of further research 
suggested by this study include the 
effect of multimedia on vocabulary 
learning and the impact of learning 
styles and feedback on computer- 
mediated learning. 

As the exploration of multime- 
dia continues, projects such as the 
one undertaken at CUA will dem- 
onstrate the feasibility of creating 
lessons tailored to the context of an 
individual department. The process 
is not a simple one, however, as it 
involves ongoing cycles of devel- 
opment, implementation, and 
evaluation. The experience of one 
project provides important feedback 
to further this process and is also 
crucial in developing a knowledge 
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base in this rapidly expanding field. 
The generally positive outcomes of 
this project, both in terms of student 
learning and student attitude, serve 
to further motivate continued re- 
finement and use of the lessons at 
CUA. The questions raised suggest 
areas for much-needed future re- 
search, in particular, the need to un- 
derstand better how learners make 
use of resources afforded by the 
computer in order to manage their 
own learning. It is through contin- 
ued cycles of this development- 
implementation-evaluation process 
that teachers and students will gain 
the ability to use the tools that tech- 
nology provides more knowledgea- 
bly and effectively. 
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NOTES 

1 For information on the Libra 
authoring system, contact Dr. 
Robert Fischer, Department of 



Modem Languages, Southwest 
Texas State University, San Mar- 
cos, Texas 78666, or see 
http://www.libra.swt.edu. 

2 An IBM compatible version of 
Libra is currently being devel- 
oped. 

3 The most recent version of Libra, 
1.2.9, has a tracking log. 

4 Gilberte Furstenberg, author of 
the simulation program A la ren- 
contre de Philippe, expressed this 
view in response to my question 
at the Tech Talk session at the 
Northeast Conference on the 
Teaching of Foreign Languages, 
April, 1997. 

5 Judith Liskin-Gasparro offered 
this consideration at the Univer- 
sity of Texas Spanish Second 
Language Acquisition Sympo- 
sium, October 1997. 
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APPENDIX— QUESTIONNAIRE SPANISH 102 

This questionnaire has been developed to solicit your views about the Desti- 
nos Libra computer lessons you have just completed in order to better under- 
stand their effectiveness and how students can best make use of them. Your 
responses are needed and appreciated. Your name is requested here in order 
to give you attendance credit for Leccion 33; your responses themselves will 
not affect your grade in any way. 

Name Instructor: 



YOUR EXPERIENCE USING THE LIBRA COMPUTER PROGRAM 
Fill in the information as appropriate: 



1. How much time did you spend doing the Libra lessons? 



Lesson # 


Time spent 


Did not 
do this 
lesson 


less than 
15 min. 


15-30 min. 


30-45 min. 


45-60 min. 


more than 
1 hr. 


Lee. 22 














Lee. 30 














Lee. 33 















2. In doing the computer lessons, which of the following features did you 
use? How helpful were they? 



Features 


never 

used 


some- 

times 


often 

used 


not 

helpful 


helpful 


very 

helpful 


no 

opinion 


Directions and sample 
questions in English 
















Text map with synopsis 
















On the video palette: 
















e la y 
















fast forward 
















rewind 
















pause /stop 
















Script (not available in 
Lesson 30) 
















Glossary (not available 
in Lesson 30) 
















Feedback button 
















Grammar exercises 

















Please comment on your experiences using any of the above features: 



How useful were the computer lessons in helping you develop skills in 
the following areas? In the last column, number them 1 to 5 to rank the 
areas according to which ones were most improved for you by doing the 



O 
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computer lessons, with 1 being the least improvement up to 5 for the 
greatest improvement. 



Skills 


not use- 
ful 


somewhat 

useful 


very use- 
ful 


no opin- 
ion 


RANK 

1-5 


listening comprehension 












vocabulary learning 












cultural understanding 












situational language use 












grammar usage 













DO YOU AGREE OR DISAGREE? 

For the following statements, indicate your view by circling 5 for Strongly 
Agree, 4 for Agree, 3 for Neutral, 2 for Disagree, and 1 for Strongly Disagree. 
Please clarify your response in the far right hand margin as needed. 



Statement 



SA AND SD Comments 



1. I am generally comfortable working on computers. 5 4 

2. I had sufficient orientation to the Libra program before 5 4 

doing the Libra lessons on my own. 

3. I cannot follow the Destinos storyline just by watching 5 4 

the segments in the Libra lesson. 

4. The computer lessons are sufficiently reinforced in class 5 4 

activities. 

5. I often feel frustrated while doing the lesson. 5 4 

6. At the beginning of this semester, my attitude toward 5 4 



using computers to leam a foreign language was positive. 

7. After completing the Libra computer lessons, my attitude 5 4 

about using computers to leam a foreign language is 

positive. 

8. The directions and questions in Spanish are clear so I know 5 4 

what I need to do to answer. 

9. It is difficult to "navigate" from one part of the lesson to 5 4 

another. 

10. It is easy to quit the Libra lesson and open the grammar. 5 4 

11. Doing the computer lessons helps me do better on the 5 4 

tests. 

12. The lab attendants (work study students) are helpful to 





3 2 1 
3 2 1 

3 2 1 

3 2 1 

3 2 1 
3 2 1 

3 2 1 

3 2 1 

3 2 1 

3 2 1 
3 2 1 



5 



4 3 2 1 
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me when I use the Libra computer program. 

13. It is easier to do dictations in class/ on exams after 5 4 3 2 1 

completing the Libra lesson. 

14. I would like to do more Libra computer lessons. 5 4 3 2 1 



YOUR IDEAS, PLEASE 

1. What was the main benefit you derived from doing the Libra computer 
lessons? 

What strategies did you use to maximize this benefit? 

2. For Lesson 30, you saw the complete video in lab and you did the Libra les- 
son in the MLC. How would you compare these two experiences? Explain 
the advantages and disadvantages of each. 

3. What did you like the most about doing the Libra computer lessons? 

4. If you could improve one aspect of the Libra computer lesson, what would 
it be? 

5. There are approximately 17 Destinos episodes per semester. In your opin- 
ion, what would be the ideal number of Libra computer lessons to do? 
How much time would you be willing to spend on each? Please explain. 
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